701 



An Examination of Qualitative Empirical Studies at the AHRD from 1999-2003: Research 
Purpose, Research Questions, and Inquiry Literature Cited 



Maria S. Plakhotnik and Tonette S. Rocco 
Florida International University 

Howard McCarley 
Club Med 

Silvana laninska and Judith D. Bernier 
Florida International University 

The paper examines the nature of qualitative empirical studies published in the AHRD proceedings from 
1999-2003 and discusses findings on research purpose, research question(s), and inquiry literature cited. 

Keywords: Qualitative Methodology, Researeh on Researeh, Empirieal Researeh Methods 

“Researeh is a process having a speeifie type of outeome. The outcome of research is new knowledge, obtained 
through an orderly, investigative process ” (Swanson, 1997 p. 10). 

Donovan and Marsiek (2000) doeumented three trends in human resouree development (HRD) researeh: (a) 
HRD has made strong inroads as an area of professional praetiee, (b) the field eontinues to use qualitative and 
quantitative tools relatively equally, and (e) the number of artieles published in the field inereased by 50% from 
1997 to 1998. The seeond point that qualitative and quantitative tools are used relatively equally eontradiets personal 
experienee as qualitative methods editor for HRDQ (Roeeo, 2003). Pereeptions within the field of HRD appear to be 
that (a) qualitative researeh is not published beeause qualitative methods are not honored by the editors, reviewers, 
or the field and (b) qualitative researeh laeks rigor and is therefore not of publishable quality. The questions driving 
this study are: Is qualitative researeh done in an “orderly, investigative” proeess, with rigor? Do HRD researehers 
use qualitative and quantitative methods equally? To answer the first questions we asked additional questions: 

• What information on design deeisions is evident in the methods seetions? 

• Are the questions what, how, and why adequately addressed in method seetions? 

• How do researehers that use qualitative methods deseribe and diseuss these methods? 

• What inquiry literature is used to inform design deeisions? 

The next seetion will present a brief summary of researeh on researeh in the field, followed by the researeh 
design, findings, and implieations. 

Summary of Research on Research 

As a field that has grown over the past 20 years, researeh on researeh beeomes more important as a gauge of our 
future (Williams, Bartlett, Kotrlik, & Higgins, 2002). Four approaehes have been taken to estimate HRD’s progress 
as a field through researeh on researeh: eomparison to established fields (Williams, 2001), faeulty produetivity 
through artiele publieation (Williams et ah, 2002), field produetivity through publieation outlets (Sleezer, Sleezer, & 
Paee, 1996; Dooley, 2002), and analysis of types and tools of empirieal researeh (Arnold, 1996; Hardy, 1999; van 
Hoof & Mulder, 1997; Hixon & MeClemon, 1999). 

Williams (2001) eondueted a review of researeh methods to determine if the field of HRD was following a 
developmental pattern similar to management seienee. To determine growth, artieles published in Human Resource 
Development Quarterly (HRDQ) from 1995 to 1999 and in Academy of Management Journal from 1975 to 1979 

were eompared beeause these periods represent the 20-year mark for the fields of HRD and management, 
respeetively. Artieles were searehed for “every mentioned statistieal and researeh methodology” (34-3, p. 2). Even 
though she elearly states an interest in statistieal methods only, her analysis notes the use of qualitative methods, but 
does not explore or honor qualitative methods as important to building the field. 

Researeh publieations and outlets have inereased with little investigation of the faetors assoeiated with 
inereased faeulty produetivity. On average 2 artieles were published over a five year period by 41 of 48 faeulty 
surveyed (Williams et ah, 2002). No data were eolleeted on type of artiele, other types of publieation, presentations. 
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or research methods used. An understanding exists among researchers that qualitative empirical studies take longer 
to complete and are more difficult to write up, which could impact productivity. Another way to examine the growth 
of the field is to examine the publication outlets. Sleezer et al. (1996) found 1290 refereed articles published in 258 
distinct journals between 1980 and 1994. Dooley (2002) replicated the Sleezer et al. study for the years 1995 to 2000 
finding 540 articles in 149 different journals, an 18% increase. Neither of these studies investigated the type of 
article or research method used. 

Arnold (1996) explored the state of research by analyzing AHRD conference papers for 1994 and 1995 using 
four types of research: “library research/speculative, descriptive case study or field study, field or laboratory 
experiment, [and] theoretical model or instrument construction” (p. 818). Field studies were further broken down 
“into the type of tools” used, quantitative, and qualitative. Forty-one papers used quantitative methods and 16 papers 
used qualitative methods. Four of these qualitative studies did not specify method and the remaining papers used 10 
different methods. 

Van Floof and Mulder (1997) analyzed “the contents and characteristics of research studies” (p. 1 1) published in 
the 1996 AHRD proceedings. Papers were divided into four content categories: individual development, career 
development, organizational development, and field development. Research characteristics included seven variables: 
research purpose, research strategy, research methods, data format, sampling, analysis procedure, and conclusions. 
Research methods are data collection tools such as questionnaire or interview while data format included 
quantitative, qualitative and quantitative, and qualitative, van Hoof and Mulder state that 57.7% of the studies used 
only qualitative data. This is inconsistent with other statements they make about 31.9% of papers gathering only 
narrative data “without any quantitative analysis” (p. 16) while 28.5% “quantify their qualitative data by rating 
them” (p. 16). The definitions of qualitative, quantitative, and mixed methods studies used by van Hoof and Mulder 
are not consistent with definitions accepted by methodologists in qualitative (Denzin & Lincoln, 2000) or mixed 
methods fields (Tashakkori & Teddlie, 1998). 

Hixon and McClemon (1999) followed but did not replicate the Sleezer et al. (1996) search strategy, finding 66 
articles in 23 journals published in 1997. The articles were classified by subject, research method, research 
participants and publication venue. Research method was divided into four types of research and “two tools (i.e., 
qualitative and quantitative)” (p. 899) and duplicated Arnold’s (1996) classification scheme. All of the articles and 
papers were placed in two categories— qualitative or quantitative and ignored the possibility of a category of mixed 
methods. Twenty-one of the sixty-six articles relied on qualitative tools. The 45 quantitative articles were broken 
down by method of analysis but not the qualitative articles. 

Hardy (1999) examined the methodological appropriateness of papers presented at the 1997 and 1998 AHRD 
conference. After setting the stage for the inclusion of mixed method studies through his discussion of hybrid 
designs, his findings are only discussed in terms of the dichotomy of qualitative/quantitative categories instead of 
the continuum possible when using hybrid designs or mixed methods. For Hardy, the role of qualitative methods is 
to develop new theory, expand conceptual frameworks, and enhance understanding of social realities while 
quantitative methods are used to test and generalize theory. Hardy states the ultimate usefulness of hybrid designs is 
dependent on the comparative relationships of the results, conclusions, and contributions to the field, yet he 
advocates only the use of multivariate studies to mature the field. 

What has been lacking in these studies of research in HRD is a systematic analysis of mixed methods and 
qualitative empirical studies. The perception of qualitative research not being honored is evident by the uneven 
treatment given to qualitative studies, if they are included in the data at all in these studies on research. 

Research Design 

The research team is composed of a faculty member and five graduate students. We set out to examine the nature of 
qualitative empirical studies over the past five years by searching AHRD proceedings from 1999-2003. The first 
step was to determine how many articles were qualitative empirical studies, no mixed methods or mixed model 
designs (Tashakkori & Teddlie, 1998) were included. Of 695 total papers presented in these five years, less than 
25% or 173 papers were considered qualitative studies. Titles, abstracts, and methods sections were scanned 
searching for words such as qualitative study or the specific method used. During the data collection and analysis 
this number was further reduced to 151 (22%). During the two-step data collection phase papers were read more 
thoroughly in order find the data to be input into the ACCESS database, read again by the second researcher 
checking the data, and again by the third researcher who analyzed specific categories. 

We met regularly to determine categories, to ensure consistency in search strategies, use of the database, and to 
discuss other issues as they occurred. Data was collected in these categories: 1) method, 2) rationale for method, 3) 
research question/purpose/propositions, 4) participants, 5) sample, 6) data collection, 7) data analysis, 8) data 
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management, 9) integrity measures, and 10) inquiry literature used. Patton (2002) served as our baseline for eategory 
definitions and for what should be ineluded in a eategory. The data eolleetion proeess oeeurred in two steps. An 
original researeher went through a year’s worth of qualitative papers inputting the data into the Aeeess database 
ereated for this purpose. A seeond researeher went through the same papers to eheek the data. The entire group 
diseussed any diserepaneies found by the seeond researeher until agreement was reaehed. After eaeh step additional 
papers were eliminated as not being qualitative studies. However, even if the study used mixed methods, it was 
ineluded if the author stated it was a qualitative study. 

The fourth step was to analyze the data using eontent analysis proeedures where, in Patton’s, words we 
attempted “to identify eore eonsisteneies and meanings” (p. 453). Two questions were used as a guide: How is 
qualitative researeh performed and reported by HRD praetitioners and researehers today? To what extent does 
Patton’s (2002) taxonomy of qualitative researeh methods eorrespond to methods reported by HRD praetitioners and 
researehers? When the data eolleeted did not fit into Patton’s (2002) taxonomies, themes that emerged from the data 
were used. A primary researeher was responsible for the preliminary analysis of speeifie eategories. 

Discussion of Findings 

Due to spaee limitations only findings from the eategories researeh purpose, researeh question(s), and inquiry 
literature used will be diseussed. 

Research Purpose 

Different researeh purposes refleet researehers’ ehoiees of theoretieal traditions and values and “lead to 
different ways to eoneeptualizing problems, different designs, different types of data gathering, and different ways 
of publieizing and disseminating findings” (Patton, 2002, p. 222). Patton identifies five types of purposes in 
qualitative researeh. Basie researeh aims to understand and explain fundamental issues of a diseipline and to 
generate new theories or test the existing ones. Applied researeh generates knowledge to help people solve a real- 
world problem. Summative evaluation “judge[s] overall effeetiveness to inform major deeisions about whether the 
program should eontinue” and formative evaluation “aim[s] to improve programs” (p. 218). Aetion researeh 
addresses problems of a program or organization, without an intention to generalize the knowledge to other eontexts. 
Eighty authors (53%) eondueted applied and sixty-six (43.7%) eondueted basie researeh (see Table A). Four studies, 
less than 3%, identified their purpose as aetion researeh (e.g., Bailey & Hahn, 1999; Bierema, 2002). Only Parker 
and Thomas (2003) deseribe their researeh purpose as “qualitative evaluation” whieh is “both a formative and a 
summative evaluation” (p. 168). 



Table A. Examination of Research Purpose through Patton ’s (2002) Typology 



Year 


Total 

Papers 


Basic 


Percent 


Applied 


Percent 


Evaluation 


Percent 


Action 

Research 


Percent 


1999 


46 


14 


30.4 


31 


67.4 


0 


0 


1 


2.2 


2000 


25 


8 


32 


17 


68 


0 


0 


0 


0 


2001 


31 


17 


54.8 


12 


38.7 


0 


0 


2 


6.5 


2002 


30 


20 


66.7 


9 


30 


0 


0 


1 


3.3 


2003 


19 


7 


36.8 


11 


57.9 


1 


5.3 


0 


0 


Total 


151 


66 


43.7 


80 


53 


1 


.66 


4 


2.6 



Sinee “purpose is the eontrolling foree in researeh” (Patton, 2002, p. 213), purpose statement beeomes “the 
most important statement in an entire researeh study” (Creswell, 2003, p. 87). Out of 151 papers, 138 (91.4%) 
provided a purpose statement (see Table B). Of the remaining thirteen papers, twelve ineluded researeh questions 
(e.g., Huhta & Niemi, 2001; Kehrhahn 1999) and one - propositions (Tate-Blake, 2000). 

“A good qualitative purpose statement eontains important elements of qualitative researeh, uses researeh words 
drawn from literature of that inquiry . . ., and employs the proeedures of an emerging design based on experienees of 
individuals in a natural setting” (Creswell, 2003, p. 88). Most authors, 132 (87.4%), provided a elear purpose 
statement that was easy to identify and understand. For example, Bierema (2002) writes, “The purpose of this 
researeh was to study women’s gender eonseiousness development through an aetion researeh projeet with a group 
of eight women, the majority of who are employed as domestie violenee prevention and treatment poliey makers and 
serviee providers in the Midwest” (paper 7-2). This purpose statement starts with a signaling phrase and ineludes the 
phenomena under investigation, researeh design, the partieipants, and geographie loeation. 
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Table B. Research Purpose Statement by Year 



Year 


Total Papers 


Total Purpose 


Pereent 


Clear Purpose 


Pereent 


1999 


46 


43 


93.5 


41 


89 


2000 


25 


20 


80 


20 


80 


2001 


31 


27 


87 


27 


87 


2002 


30 


30 


100 


26 


86.7 


2003 


19 


18 


94.7 


18 


94.7 


Total 


151 


138 


91.4 


132 


87.4 



However, six studies expressed their purpose in other terms. Four authors framed their researeh purpose as a 
paper purpose whieh is sometimes reserved for eoneeptual pieees rather than empirieal studies. For example, 
Boudreaux, Chermaek, Lowe, Wilson, and Holton (2002) state, “the purpose of this paper is to deseribe the 
qualitative projeet and the androgogieal proeess undertaken” and deseribe the projeet outeome instead of its purpose 
-“the ereation of a database that would be available for future researeh and seholarly study” (paper 35-2). Two other 
authors framed researeh purpose as a researeh problem. For example, Bourland (1999) writes, “Researeh problem 
was therefore defined to identify how existing international HRD experienees and praetiees at the ONS eompared 
with best praetiee in eross-eultural eonsultaney, and to draw eonelusions about the ehanges needed to those praetiees 
and the implieations for advaneing theories” (p. 697). The purpose statement is a separate element of a study and 
should not be eonfused for with a problem statement. Aeeording to Creswell (2003), the purpose “builds on a need 
(the problem) and is refined into speeifie questions (the researeh questions)” (Creswell, 2003, p. 88). While the 
problem statement identifies a gap in knowledge, the purpose statement deseribes “what needs to be done” to 
address the gap (Merriam & Simpson, 2000, p. 19). 

Research Questions 

Researeh questions serve as another “signpost” of the researeh plan, follow the purpose statement, and elarify 
and narrow the foeus (Creswell, 2003, p. 105). They “begin to operationalize the eoneeptual framework”, elarify 
theoretieal assumptions, prioritize issues of interest, foeus data eolleetion, help with deeisions about sampling and 
data eolleetion tools, and, henee, “eonstrain the possible types of analysis” (Miles & Huberman, 1994, p. 22). One 
hundred twenty-nine papers (85.4%) provided researeh questions. Researeh questions should “assume two forms: a 
eentral question and assoeiated subquestions” (Creswell, 2003, p. 105). Trying to answer too many questions of 
either form ean eause the researeher to “easily lose the forest for the trees and fragment the eolleetion of data . . . 
[making] it harder to see emergent links aeross different parts of the database and to integrate findings” (Miles & 
Huberman, 1994, p. 25). Out of the 129, only 12 studies (9.3%) followed this division. Ten of these 12 studies used 
one eentral question followed by two (e.g., Roeeo & Stein, 2000) to eleven (e.g., Kraemer, 2000) subquestions. 
O’Neil (2001) used two eentral questions followed by five subquestions, whieh were assoeiated with both eentral 
questions as a set. Only Yoon (2003) formulated 3 eentral questions eaeh followed by a set of subquestions. 

One hundred three studies (79.8%) ehose to rely only on eentral questions (see Table C). Less than 15% of the 
researehers used one question, 17% used two eentral questions, one third (29.5%) used three eentral researeh 
questions, and almost 19% used more than three (18.6%). 

Fourteen studies (10.9%) used other ways to narrow the purpose. Researeh questions were grouped by “four 
areas” (Winterton & Winterton, 2002) and by “four eomponents” of the issue of inquiry (Raspiller, 1999). Four 
studies provided only examples of researeh questions. For instanee, Ravishankar (1999) writes, “some of the 
researeh questions we attempted to answer were the following” (p. 92) and provides half a dozen questions. Though 
qualitative researehers should not state objeetives, goals, or hypotheses (Creswell, 2003), two studies were 
answering their researeh questions and testing hypotheses (Sewe, 1999; Walton & Martin, 2000) and two more were 
answering questions and aehieving objeetives (Kamau, MeLean, & Ardiehvili, 1999; O’Neil, 1999). Four other 
authors ineluded either two different sets of questions in the paper (e.g., Polaeh, 2001) or a set of questions “to 
lessen the eonfusion” in addition to the eentral question and subquestions (Lien, 2000). 



Table C. The Use of Research Questions 








Research Questions 




Other 






Central questions only 




Central questions 
and subquestions 


Year 


One 


Two 


Three 


More than three 


1999 


8 


5 


11 


5 


3 


9 


2000 


2 


2 


7 


5 


3 


2 


2001 


4 


5 


6 


6 


3 


2 


2002 


4 


4 


7 


5 


2 


1 
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Table C. The Use of Research Questions (Con’t) 



Year 


One 


Two 


Three 


More than three 


Central questions 
and subquestions 


Other 


2003 


1 


6 


7 


3 


1 


0 


Total 


19 


22 


38 


24 


12 


14 


Percent 


14.7 


17 


29.5 


18.6 


9.3 


10.9 



Implications for the Field Regarding Research Purpose and Research Questions 

All researchers provided either a purpose statement or research questions. However, since “the first step in a 
research process is getting clear about purpose” (Patton, 2002, p. 213), it is what is missing or confusing in the 
papers that deserves more attention. Twenty- four studies (16%) lacked either the purpose statement or research 
questions. If some AHRD qualitative researchers approach research without a clear purpose, the possibility exists 
that this will undermine the credibility of the findings, qualitative research in general, and the Academy, which 
strives toward a vision of “leading human resource development profession through research” (AHRD, 2005). Our 
finding that qualitative AHRD researchers prefer conducting applied research over basic supports Chalofsky’s 
(2004) conclusion that HRD “is still primarily practice-driven” (p. 425). Research and theory building did not have 
supports for dissemination until fairly recently with the establishment of the Academy, HRDQ and Performance 
Improvement Quarterly, so “we have had practitioners’ and consultants’ educated opinions and ideas cloaked as new 
theories and/or approaches driving practice” (p. 426, original italics). Between 1999-2003, the number of 
qualitative applied research studies exceeded basic research studies only by 14 (10%) which might indicate that 
HRD researchers are engaging in scholarly research to build a theoretical foundation for the discipline and use the 
Academy as its dissemination outlet. 

Inquiry Literature Cited 

To address the research sub-question, “What inquiry literature is used to inform design decisions?” we explored 
how many and what inquiry literature sources were used in the method sections of the manuscripts. Out of 151 
manuscripts, only three quarters (74.2%) cited inquiry literature in the method section (See Table D). Most authors 
(30%) cited either three or four inquiry literature sources; fewer (23.2%) cited two sources. Two authors (Kupritz, 
1999; Sherlock, 2002) used seven sources and one (Kupritz, 2002) used eight, the highest number of inquiry sources 
cited. 

Table D. Analysis of Inquiry Literature by Number and Year 



Number of inquiry literature sources cited 



Number of 



Year 


papers 


0 


% 


1 


% 


2 


% 


3-4 


% 


5-8 


% 


1999 


46 


11 


23.9 


3 


6.5 


8 


17.4 


18 


39 


6 


13 


2000 


25 


6 


24 


3 


12 


9 


36 


6 


24 


1 


4 


2001 


31 


12 


38.7 


7 


22.6 


5 


16.1 


7 


22.6 


0 


0 


2002 


30 


8 


26.7 


2 


6.6 


5 


16.6 


8 


26.7 


7 


23.3 


2003 


19 


1 


5.3 


3 


15.8 


8 


42.1 


6 


31.6 


1 


5.3 


Total 


151 


38 


25.2 


18 


11.9 


35 


23.2 


45 


30 


15 


10 



Six inquiry literature sources were used most frequently. Of 113 authors that cited inquiry literature, 21 (18.6%) 
used Naturalistic Inquiry (Lincoln & Guba, 1985), 20 (17.7%) cited Patton’s (1990, 2002) Qualitative Evaluation 
and Research Methods, and 19 (16.8%) referred to Miles and Huberman’s (1994) Qualitative Data Analysis. Twelve 
percent of authors cited Yin (1994), nine percent - Meriam (1998), and eight percent - Creswell (1998). 

Implications for the Field Regarding Inquiry Literature Cited 

A high level of scholarship is demonstrated through researchers’ familiarity with research methodology and 
techniques which allow researchers to ground their research, present sound arguments, and contribute to their 
disciplines (Hart, 1998). In manuscripts, researchers demonstrate this understanding in the Method section by 
explaining “what was done, how it was done, and why it was done, and answers to these questions should be 
grounded in the inquiry literature” (Rocco, 2005, p. 15). Citing sources “whenever possible” to support statements is 
also a requirement of the APA Manual (2001, p. 28), the guiding source for publications in the Academy. Since the 
AHRD consists primarily of university faculty, the fact that over half of them cited between two and four sources 
does not come as a surprise. However, the quarter of the papers not grounded in the inquiry literature is alarming. 
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The amount of inquiry literature tripled between 1984 and 1994 (Miles & Huberman, 1994). Sinee defeets in 
researeh design and management are “a major eause for the rejeetion of manuseripts” (APA, 2001, p. 4), this 
disregard for inquiry literature undermines the quality of qualitative researeh and its potential for advaneing HRD. 

Conclusion 

As a field, if we do not eoneem ourselves with the quality of researeh design in qualitative empirieal studies we 
deerease the ehanees of publieation of this work. This limits the potential impaet the researeh might have had on the 
field. For a field eoneemed with theory development (Lynham, 2000), with praetiee informing theory and theory 
informing praetiee (Jaeobs, 1997; Jarvis, 1999), this is disturbing. 

The researeh proeess ineludes “the very important final step of reporting the results” (Merriam & Simpson, 
2000, p. 173) whieh involves being able to write up the researeh as a eonferenee paper and then manuseript. If 
qualitative empirieal researeh aims to eontribute to a field, it must be presented in a logieal and eoherent manner that 
praetitioners understand and other researehers ean use. In order for this to happen, we must teaeh our students, 
noviee seholars, and praetitioner-researehers how to present their ideas well (Jarvis, 1999) and how to present the 
researeh design elearly and eoneisely (Roeeo, 2003). Sinee “researeh is eentral to the development of any field of 
study” (Merriam & Simpson, 2000, p. 1) when researeh about praetieal work based problems goes unpublished, the 
potential eontribution to the field is lost (Jarvis, 1999). 
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